PME-NA 2011 Proceedings 372 


RESEARCH IN MATHEMATICS EDUCATIONAL TECHNOLOGY: 
CURRENT TRENDS AND FUTRUE DEMANDS 


Shannon O. Driskell Robert N. Ronau Christopher R. Rakes 
University of Dayton University of Louisville Institute of Education Sciences 
SDriskell@udayton.edu Bob@louisville.edu Christopher.Rakes@gmail.com 
Sarah B. Bush Margaret L. Niess David Pugalee 
Bellarmine University Oregon State University University of North Carolina- 
sbush@bellarmine.edu niessm@onid.orst.edu Charlotte 


David.Pugalee@uncc.edu 


This systematic review of mathematics educational technology literature identified 1356 
manuscripts addressing the integration of educational technology into mathematics instruction. 
The manuscripts were analyzed using three frameworks (Research Design, Teacher Knowledge, 
and TPACK) and three supplementary lenses (Data Sources, Outcomes, and NCTM Principles) 
to produce a database to support future research syntheses and meta-analyses. Preliminary 
analyses of student and teacher outcomes (e.g., knowledge, cognition, affect, and performance) 
suggest that the effects of incorporating graphing calculator and dynamic geometry technologies 
have been abundantly studied; however, the usefulness of the results was often limited by missing 
information regarding measures of validity, reliability, and/or trustworthiness. 


Educational technology (i.e., digital technology, as opposed to other forms of educational 
tools such as overhead projectors or physical manipulatives) is promoted to mathematics teachers 
as a research-based strategy for improving student outcomes. Although research on mathematics 
educational technology appears at first glance to be ubiquitous, the usefulness of this research to 
practitioners and researchers is limited by lack of attention to research design and validity, 
reliability, and threats to validity (Rakes et al., 2011). Additionally, much of the research appears 
to be unorganized, with topics such as graphing calculators studied often, while other topics such 
as virtual manipulatives understudied (Ronau et al., 2010). The purposes of this systematic 
review were to (1) examine the evidence of technology impact on the teaching and learning 
mathematics in K-13, graduate, teacher development, and adult education using three 
frameworks (Comprehensive Framework of Teacher Knowledge [CFTK], Research Design, and 
Technology, Pedagogy, and Content Knowledge [TPACK]) and three supplementary lenses 
(Data Sources, Outcomes, and NCTM Principles) and (2) assess the utility of each framework for 
guiding the synthesis of mathematics educational technology research. 


Theoretical Framework and Background 

Three frameworks were applied to the analysis in the set of mathematics educational 
technology studies discovered by the systematic review: Research Design, CFTK, and TPACK. 
The research design framework was used to guide the investigation of the types of research 
approaches used in mathematics educational technology research. The complex nature of 
questions pertaining to educational technology effectiveness requires a variety of research 
designs such as (1) experimental or quasi-experimental studies, (2) large-scale studies, (3) 
studies with sufficient statistical information to be included in meta analysis and mixed- 
methodology studies, (4) studies with rich analysis of student content knowledge, and (5) studies 
that address the complexities of learners, classrooms, and schools (Bell, Schrum, & Thompson, 
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2008; Means, Wagner, Haertel, & Javitz, 2003). However, without explicit attention to the 
alignment of the research design to the questions of interest, the validity and reliability of the 
measures used, and the threats to validity within the chosen design, the reported outcomes will 
be less likely to have been founded on scientific principles or to be replicable (Shadish, Cook, & 
Campbell, 2002). The usefulness of such studies to practitioners and researchers will be, 
therefore, limited without robust attention to research design issues. The Research Design 
framework (Ronau et al., 2010) was compiled from several sources to address pertinent issues 
across a wide range of research types (e.g., Creswell, 2009; Shadish et al., 2002; Shavelson & 
Towne, 2002; Teddlie & Tashakorri, 2009). For a detailed description of the Research Design 
framework, see Rakes, Wagener, and Ronau (2010). 

Within the past few years, two new teacher knowledge frameworks have been proposed that 
have the potential to support the research community in responding to questions on the impact of 
technology on learning. The Comprehensive Framework of Teacher Knowledge (CFTK; Figure 
1) provides a three-dimensional model for addressing multiple aspects of teacher knowledge and 
their interactions (Rakes, Ronau, & Niess, 2010; Ronau, Rakes, Wagener, & Dougherty, 2009; 
Ronau, Wagener, & Rakes, 2009). This model transforms current understanding of teacher 
knowledge from a linear structure to a three dimensional model, as shown in Figure 1, by pairing 
six inter-related aspects into three orthogonal axes: 1) Field, comprised of Subject Matter and 
Pedagogy; 2) Mode, comprised of Orientation and Discernment; and 3) Context, comprised of 
Individual and Environment. For a detailed description of the CFTK aspects and dimensions, see 
Ronau and Rakes (in press). 
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Figure 2. CFTK framework of teacher knowledge as a three-dimensional 
structure. 


The Technology, Pedagogy, and Content Knowledge (TPACK) framework defines the 
knowledge needed by teachers to integrate technology into the pedagogy of particular subject 
matter (e.g., Mishra & Koehler, 2006; Niess, 2005). In its entirety, TPACK consists of a set of 
descriptive knowledge components embedded in an educational Context, Content Knowledge 
(CK), Pedagogical Knowledge (PK), and Technological Knowledge (TK), and a series of 
interactions, Pedagogical Content Knowledge (PCK), Technological Pedagogical Knowledge 
(TPK), Technological Content Knowledge (TCK), and TPACK. The initial TRACK framework 
has been extended to provide benchmarks of the development of this knowledge as shown in 
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Figure 2, including recognizing, accepting, adapting, exploring, and advancing (Niess, Lee, & 
Sadri, 2007). 
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Figure 3. Model of teacher thinking and understanding as that knowledge develops 
toward the intersection identified as important by TPACK. 


Based on feedback through peer debriefing (Rakes et al., 2010), three additional lenses (Data 
Sources, Outcomes, and NCTM Principles) were added to provide a more comprehensive 
snapshot of the research landscape, guiding practitioners to choose best practices and for guiding 
future research directions. 

Using these frameworks and lenses, we began our investigation with two overall questions: 
(1) To what degree do the three frameworks, Research Design, CFTK, and TPACK, capture the 
scope of mathematics educational technology research? (2) What Data Sources, Outcomes, and 
NCTM Principles are addressed in mathematics educational technology research? To what 
degree, and how, implicit/explicit? 


Method 

A research synthesis (Cooper & Hedges, 2009) was conducted to address the two overall 
questions. To identify the most representative sample that was relevant to the questions of 
interest (i.e., construct validity), a wide array of databases were searched using terms to restrict 
the sample based on three inclusion criteria: (1) The study needed to examine a technology-based 
intervention; (2) The intervention needed to target the learning of a mathematics concept or 
procedure; (3) The manuscript needed to be available in the English language. The database 
platforms and individual databases included EBSCOWeb (ERIC, Academic Search Premier, 
PsychInfo, Primary Search Plus, Middle Search Plus, Educational Administration Abstracts), 
JSTOR (limited to the following disciplines: Education, Mathematics, Psychology, and 
Statistics), OVID, ProQuest (Research Library, Dissertations & Theses, Career & Technical 
Education), and H. W. Wilson Web (Education Full Text). From these databases, 1356 
manuscripts (journal articles, book chapters, technical reports, conference proceedings, master's 
theses, and doctoral dissertations) were identified as being potentially relevant to the questions of 
interest. 

The initial coding database was pilot tested with three articles and two coders to help refine 
the coding database. Refinements based on the results of this pilot test were examined with all 
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six researchers coding the same, original three articles. This process was repeated through three 
more iterations of refinement and coding of 27 more articles (1.e., 30 articles were group coded). 

After the analysis of 473 manuscripts, a number of coding issues emerged that required 
attention. Extensive team discussions led to a number of coding clarifications to improve team 
alignment, including how to: code studies with meta-analyses and/or systematic reviews, mixed 
methodology designs and single subject designs, action research, and survey research; code 
purposive and convenience sampling, subject dialog data, and modified and validated 
instruments; and record evaluative comments when deemed necessary. Finally, a number of 
coding form issues was addressed. The existing database of 473 studies was aligned with this 
new set of procedures and understandings, and the team developed a new process of coding the 
remaining studies that paired each of the six coders with all the other coders to provide a mixed 
set of double coding for the remainder of the studies. The new coding design created a 
completely counter-balanced design with all six coders that provided greater inter-rater reliability 
and content validity of the coding. With this plan, every manuscript was coded by two members 
of the coding team, and each member coded 59 studies with every other member. Any 
discrepancies between coder and re-coder were recorded and discussed by the pair and by the 
full team as needed. 


Preliminary Results 

At the time of the writing of this paper, 473 manuscripts have been coded and cross-validated 
(i.e., double coded and checked for accuracy). Twenty four of these manuscripts were screened 
out because they were not relevant to the questions of interest (i.e., did not address the learning 
of mathematics concepts and procedures, did not involve technology, or was not available in 
English), leaving 449 manuscripts in the sample. Initial results were examined by grouping the 
manuscripts into four categories (not mutually exclusive) of outcomes: Student Achievement and 
Learning; Student Orientation, Discernment, and Learning Behavior; Teacher Knowledge; and 
TPACK. The manuscripts and their 449 characteristics were analyzed through descriptive 
statistics as an initial method for interpreting the landscape of mathematics educational 
technology research. 

The Role of Educational Technology in Student Achievement and Learning 

Of the relevant manuscripts, 218 addressed educational technology in mathematics with a 
view of improving student achievement and learning. As shown in Table 1, over half (N=113) 
were dissertations and many of the remainder (N=69) were journal articles. Over half (N=118) 
were purely quantitative studies. The remaining studies were qualitative (N=36), mixed methods 
(N=43), non-research (N=9), meta analysis/systematic review (N=6), literature driven (N=5), or 
single subject (N=1). Performance assessments (e. g., tests, performance tasks, grades, GPA, 
etc.) were the most common sources of data used in these manuscripts, while journals (all types), 
focus groups, and non self-report surveys were the least used. 

The manuscripts in this subsample most commonly addressed the Algebra NCTM Content 
Standard (N=159) and the Problem Solving Process Standard (N=89). Graphing calculators 
(N=43), tutorial software (N=39), and dynamic geometry software (N=25) were the more 
regularly studied technologies. In regards to information on measures of reliability and validity 
for the quantitative and mixed methods studies, approximately 41% of the manuscripts addressed 
reliability, 30% addressed validity, and 57% addressed threats to validity. For the qualitative and 
mixed methods studies, 60% attended to trustworthiness where approximately 62% of these 
studies attended to only one form of trustworthiness. 
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Type of Non- Quali- | Quanti- | Mixed Single | Meta- Liter- | Total 
Manuscript | research | tative | tative Methods | Subject | Analysis/ | ature 

by Research Sytematic 

Design Review 

Book 0 1 1 0 0 0 1 3 
Chapter 

Conference | | 2 1 2 0 0 1 o 
Paper 

Disserta-tion | 0 15 a9 34 1 4 0 113 
Journal 8 16 35 1 3 69 
Master’s 0 2 18 1 0 0 0 21 
Thesis 

Report 0 0 4 0 0 1 0 5 
Grand 9 36 118 43 1 6 5 218 
Total 


Table 1. Student achievement and learning manuscripts by research design. 


The Impact of Educational Technology on Student Orientation 

Of the relevant manuscripts, 126 had examined the impact of educational technology on 
student orientation (1.e., the affective domain), discernment (i.e., the cognitive domain), and 
learning behaviors (e.g., student dialog and collaboration). The manuscripts consisted of 71 
dissertations and 39 journal articles. Of these, 62 used purely quantitative analyses, 28 used 
purely qualitative methodologies, and 27 used mixed methodology. Self-report orientation 
survey data and performance assessment data were the two most common types of data sources; 
the top seven data sources are listed in Table 2. The Algebra NCTM Content Standard (N=32) 
and the Problem Solving Process Standard (N=23) were the most commonly addressed NCTM 
standards. Graphing calculators (N=24) were the most common type of calculator-based 
technology, followed by non-scientific calculators (N=16). The three most frequently used 
nonweb-based software were tutorial software (N=15), dynamic geometry (N=13), and algebra 
(N=12). Distance learning stood out among the web-based technologies (N=10) as these other 
technologies were often not addressed or addressed in only one or two manuscripts. Only 27% of 
the quantitative and mixed methods studies addressed issues surrounding validity, 61% 
addressed threats to validity, and 42% addressed reliability. Of the qualitative and mixed 
methods studies, 73% attended to trustworthiness, with 55% that attended to only one form of 
trustworthiness. 


Data Sources Number of Studies 
Self-Report Orientation Survey Data 83 

Performance Assessment Data 76 

Observation Data 42 

Interview Data 37 

Content Analysis Data 28 

Self-Report Polls and Census Survey Data 13 

Researcher Journal Data 8 


Table 2. Top seven data sources for student orientation. 
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The Interaction of Teacher Knowledge Aspects in Educational Technology Research 

Teacher Knowledge was examined as an outcome in 72 manuscripts, of which 39 were 
journal articles and 26 were dissertations. The most common research design was qualitative 
(N=35), followed by non-research (N=15), mixed methods (N=11), and quantitative (N=8). The 
four most frequently used data sources were observation data (N=42), interview data (N=30), 
content analysis data (N=29), and self-report orientation survey data (N=24). Graphing 
calculators, dynamic geometry software, and spreadsheets were the most commonly studied 
technologies. The Algebra Content Standard (N=20) and the Problem Solving Process Standard 
(N=11) were the most commonly addressed NCTM standards. The distribution of CFTK aspects 
and interactions from this sample was compared to the distribution reported in Ronau and Rakes 
(in press) for teacher knowledge studies across multiple subject matter domains. The average 
number of aspects examined in mathematics educational technology (x= 1.53, SE = 0.207) 
appeared to be smaller than the average number of aspects examined across multiple subject 
matter domains (x= 2.16, SE = 0.119). This difference appeared to be statistically significant 
(tae-71=3.74). This result may indicate that the research field in mathematics education 
technology may be considering less complex perspectives of teacher knowledge than other 
subject matter domains. Reliability was attended to in 32% of the quantitative and mixed 
methods studies, with validity and threats to validity attended to in 11% and 53%, respectively. 
Trustworthiness was addressed in 65% of the qualitative and mixed methods studies, where 60% 
addressed only one form of trustworthiness. 


Graphing | Dynamic | Spread- | Graphing | Algebraic | Statistics | Tutorials 


Calculators | Geometry | sheets Software | Software | Software 
Number | 22 19 14 7 5 5 5 
of 
Studies 


Table 3. Top seven technologies for teacher knowledge. 


The Use of TPACK to Guide Educational Technology Research 

TPACK, as a guiding framework of the knowledge teachers need for integrating educational 
technology in mathematics, was employed either explicitly or implicitly, in 219 manuscripts, of 
which 102 were journal articles and 85 dissertations. Of these manuscripts, 72 used purely 
quantitative methodologies, 48 used purely qualitative methodologies, 27 used mixed 
methodologies, and 61 did not employ any type of research design (i.e., anecdotal support of 
hypotheses or descriptions of techniques). Performance assessment was the most commonly used 
data source (N=86). The NCTM Content Standards considered most often were Algebra (N=67), 
Geometry (N=45), and Number & Operations (N=37), while Problem Solving (N=35) was the 
most commonly considered Process Standard. Graphing calculators (N= 54) and dynamic 
geometry software (N=49) were the two most common types of technology used. Only 24% of 
the quantitative and mixed methods studies addressed validity issues, 81% addressed threats to 
validity, and 39% addressed reliability. Of the qualitative and mixed methods studies, 65% 
attended to trustworthiness, with 63% that attended to only one form of trustworthiness. 


Summary 
Several patterns were common among all four outcome groups. Dissertations and journal 


Wiest, L. R., & Lamberg, T. (Eds.). (2011). Proceedings of the 33rd Annual Meeting of the North 
American Chapter of the International Group for the Psychology of Mathematics Education. 
Reno, NV: University of Nevada, Reno. 


Articles published in the Proceedings are copyrighted by the authors. 


PME-NA 2011 Proceedings 379 


articles were the two most common types of manuscripts, with quantitative studies being the 
most prevalent research design among three of the four areas (all except teacher knowledge 
outcomes). Also among these three areas, performance assessment and self-report orientation 
survey were the two most often used data sources. The most common NCTM Content and 
Process Standards addressed in all four areas were Algebra and Problem Solving, respectively. 
Graphing calculators were consistently the most frequently used calculator-based technology, 
while dynamic geometry was either the most commonly used or second most commonly used 
non-web-based technology for all four areas. Web-based technologies were the least frequently 
used type of technology among all four areas. 

Missing information regarding measures of validity, reliability, and/or trustworthiness was 
prevalent. Overall, approximately 40% of the quantitative and mixed methods studies addressed 
reliability, 27% addressed validity, and 64% addressed threats to validity. Trustworthiness was 
attended to in approximately 65% of all qualitative and mixed methods studies. 


Discussion 

The completion of the study will provide a searchable database of educational technology 
studies from 1968 to 2010, containing key information organized by three frameworks and four 
lenses. With this data, we will be able to better describe the landscape of educational technology 
research in mathematics, providing significant detail about the type, quality, content, and 
alignment of the studies. Doctoral students and advisors will benefit from an analysis of the 600+ 
dissertations in the sample, providing a guide to over- and under-studied dissertation topics (e.g., 
impact of graphing calculators in algebra). The research team will be able to identify gaps in the 
research base with respect to a number of study characteristics organized by the frameworks and 
lenses described above, as well as the depth and quality of areas well-studied. Detailed coding of 
research design features will allow for rigorous examination of the evidence currently available 
to the field in the form of meta-analysis and qualitative research syntheses. The systematic nature 
of this review will provide a foundation for future investigations and replication. Additionally, 
once the initial coding is completed, the task of updating the database with newly released 
manuscripts can easily be accomplished. Finally, this study will also include an evaluation of the 
utility of each of the three frameworks and four lenses used in the analysis to capture the 
perspective and the critical details of educational technology research. 
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